HarmonyMixer: Mixing the Character of Chords among Polyphonic Audio

This paper describes HarmonyMixer, a method that enables a user without musical expertise to personalize the mood of existing polyphonic musical recordings by modifying their chord sequences. Our method lets the user choose a reference songwith a character that the user wants reflected in chords of a target song. It is, however, difficult to modify chords in existing complex sound mixtures since technologies of sound source separation and multipitch analysis are not yet accurate enough for those mixtures. To overcome this difficulty, HarmonyMixer does not rely on those technologies and instead modifies chords by leveraging chromagrams. It first analyzes a chromagram feature matrix by using Bayesian non-parametric Non-negative Matrix Factorization, and then interpolates basis matrices obtained from reference and target songs to convert the chromagram of the target song. It finally modifies the spectrogram of the target song by reflecting the difference between the original and converted chromagrams while considering relations between frequency bins and chroma bins. Listening to the output from our method confirmed that modification of chords had been derived.